Conversation
mat-hek
left a comment
mat-hek
left a comment
There was a problem hiding this comment.
Examples aren't adjusted to changes in the raw video format
.gitignore
Outdated
|
|
||
| # Created by https://www.gitignore.io/api/c,vim,linux,macos,elixir,windows,visualstudiocode | ||
| # Edit at https://www.gitignore.io/?templates=c,vim,linux,macos,elixir,windows,visualstudiocode | ||
| # Edit at https://www.gitignore.io/?yolos=c,vim,linux,macos,elixir,windows,visualstudiocode |
| @@ -0,0 +1,337 @@ | |||
| # Membrane YOLO Plugin demos | |||
|
|
|||
| ```elixir | |||
There was a problem hiding this comment.
you can add this, so there's better chance that cargo is found and Ortex compiles on Mac
| ```elixir | |
| ```elixir | |
| System.put_env("PATH", "/opt/homebrew/bin:#{System.get_env("PATH")}") | |
examples/yolo.livemd
Outdated
| ## Download fixtures and model | ||
|
|
||
| ```elixir | ||
| tmp_dir = System.tmp_dir!() |
There was a problem hiding this comment.
use a subdir, such as tmp_dir/membrane_yolo_examples
examples/yolo.livemd
Outdated
|
|
||
| classes_path = tmp_dir |> Path.join("coco_classes.json") | ||
|
|
||
| classes_json = """ |
There was a problem hiding this comment.
if we're downloading model and samples, we can download classes.json as well
examples/yolo.livemd
Outdated
|
|
||
| <!-- livebook:{"branch_parent_index":0} --> | ||
|
|
||
| ## Live objec detection on a stream from the comptuer camera |
There was a problem hiding this comment.
| ## Live objec detection on a stream from the comptuer camera | |
| ## Live object detection on a stream from the local camera |
examples/yolo.livemd
Outdated
| end | ||
| ``` | ||
|
|
||
| ... and run it. After up to few seconds a player showing the stream with bouncing boxes should appear. |
There was a problem hiding this comment.
Mention that the player window may be not visible
lib/live_filter/model_runner.ex
Outdated
| defp maybe_draw_boxes(buffer, detected_objects, state) do | ||
| case state.draw_boxes do | ||
| false -> | ||
| %Membrane.Buffer{ | ||
| buffer | ||
| | metadata: Map.put(buffer.metadata, :detected_objects, detected_objects) | ||
| } | ||
|
|
||
| draw_fun when is_function(draw_fun, 2) -> | ||
| {:ok, image} = | ||
| Membrane.RawVideo.payload_to_image(buffer.payload, state.stream_format) | ||
|
|
||
| image = draw_fun.(image, detected_objects) | ||
| {:ok, new_payload, _stream_format} = Membrane.RawVideo.image_to_payload(image) | ||
| %Membrane.Buffer{buffer | payload: new_payload} | ||
| end | ||
| end |
There was a problem hiding this comment.
This is duplicated between the offline and live filter. We can make the function public
There was a problem hiding this comment.
This part of the code is rewrited due to moving draw_detected_objects function to our codebase. There is some place for remove code duplication, but IMO after recent changes it would only make the code more complicated if we want to avoid unnecessary conversions between raw video binary and Image
lib/offline_filter.ex
Outdated
There was a problem hiding this comment.
I'm not sure if it's worth having a separate offline filter. It's just the live filter but we wait until a buffer is processed each time. We could have a mode: :live | :offline option instead
There was a problem hiding this comment.
Now I have added another option to Live Filter. The reason why I have decided to have two filters, is because the Live one has some options that won't make any sense in Offline scenario
| function from [kino_yolo](https://github.com/poeticoding/kino_yolo) | ||
| """ | ||
| ], | ||
| additional_latency: [ |
There was a problem hiding this comment.
I think it shouldn't be the responsibility of this element, we could have a separate filter that would just add latency and would be usable in all cases like this
There was a problem hiding this comment.
Hm, but TBH I don't see any advantage of using such an element beyond this scenario.
The behaviour we want to get could be done by adding an element that sleeps on start of stream for additional_latency and then realtimer after it, but figuring this out is not trivial so embedding the solution of this problem somehow in this plugin sounds like a good idea IMO
lib/utils.ex
Outdated
| # defmodule Membrane.YOLO.Utils do | ||
| # @moduledoc false | ||
|
|
||
| # @spec draw_boxes_or_update_metadata( | ||
| # buffer :: Membrane.Buffer.t(), | ||
| # detected_objects :: [map()], | ||
| # draw_boxes :: false | (Vix.Vips.Image.t(), [map()] -> Vix.Vips.Image.t()) | ||
| # ) :: Membrane.Buffer.t() | ||
|
|
||
| # def draw_boxes_or_update_metadata(buffer, detected_objects, false) do | ||
| # metadata = buffer.metadata |> Map.put(:detected_objects, detected_objects) | ||
| # %Membrane.Buffer{buffer | metadata: metadata} | ||
| # end | ||
|
|
||
| # def draw_boxes_or_update_metadata(buffer, detected_objects, draw_fun) | ||
| # when is_function(draw_fun, 2) do | ||
| # new_payload = draw_fun.(buffer.payload, detected_objects) | ||
| # %Membrane.Buffer{buffer | payload: new_payload} | ||
| # end | ||
| # end |
lib/live_filter.ex
Outdated
| The simplest way to draw boxes is to pass `KinoYOLO.Draw.draw_detected_objects/2` | ||
| function from [kino_yolo](https://github.com/poeticoding/kino_yolo) |
There was a problem hiding this comment.
AFAIR we agreed to extract this from kino_yolo and improve colour contrast
mat-hek
left a comment
mat-hek
left a comment
There was a problem hiding this comment.
As we discussed:
- Split detection and drawing into separate elements
- Get rid of the GenServer
- Consider merging live and offline elements
FelonEkonom
changed the title
2 participants
No description provided.